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The  R&D  procurement  process  should  be  subjected  to  the  same  intense 
"cost  versus  effectiveness"  questioning  that  has  been  applied  in  recent 
years  by  the  Department  of  Defense  to  the  choice  of  alternative  proposed 
weapon  systems.   Under  this  cost/effectiveness  evaluation  approach,  alter- 
native means  of  contracting  for  research  and  development  should  be  examined 
In  light  of  their  relative  benefits  and  costs  of  all  forms.   The  present 
procurement  process  controls  the  awards  of  over  $8  billion  annually  of 
government-sponsored  research  and  development  contracts  to  industry,  uni- 
versities, and  non-profit  organizations.  A  broad  view  of  the  process  in- 
cludes not  only  the  legislation,  policies,  and  procedures  that  underlie 
R&D  procurement  but  also  the  organizations  and  people,  their  attitudes, 
and  Indirect  as  well  as  direct  effects  of  the  process. 

In  making  this  analysis,  four  questions  will  be  asked: 

1.  Who  are  the  winners  of  research  and  development  contracts? 

2.  What  are  the  key  determinants  of  the  awards  to  these  winners? 
3«   Compared  with  possible  alternatives,  what  are  the  benefits  of 

the  present  R&D  procurement  process? 
4.  Compared  with  possible  alternatives,  what  are  the  costs  of  the 

present  R&D  procurement  process? 
Answers  to  these  questions  will  be  based  on  the  results  of  three  years  of 
research  by  the  author  on  the  research  and  development  procurement  process. 

Who  Wins  R&D  Awards? 

To  determine  the  characteristics  of  R&D  award  winners  brief  question- 
naires were  mailed  to  about  1100  companies  solicited  to  bid  on  one  of  45 
formally-competitive  R&D  awards  issued  by  a  Department  of  Defense  contract- 


ing  center.   The  awards  were  for  unclassified  work,  each  involving  expend- 
itures of  between  $100,000  and  $2  million,  the  contracts  issued  between 
May,  1962  and  June,  1964.   Their  selection  for  the  sample  study  was  designed 
to  avoid  bias  by  contract  size,  technology,  or  organization  source  within 
the  DOD  installation.   Usable  replies  have  been  received  on  about  forty 
per  cent  of  the  questionnaires,  including  approximately  55  per  cent  of 
the  award  winners. 

At  this  point  in  the  cost/effectiveness  evaluation,  the  questionnaire 
data  will  be  examined  to  point  out  characteristics  of  the  winning  firms 
relative  to  their  losing  competitors.   The  data  presentation  will  follow 
a  time  sequence  starting  with  the  environment  preceeding  the  government  s 
mailing  of  Requests  for  Proposals  to  industry  and  finishing  with  submission 
of  the  R&D  proposals. 

The  first  historical  characteristic  of  the  active  competitors  for  the 
awards  is  their  prior  technical  experience.   The  questionnaires  asked: 
"Had  your  firm  performed  contract  work  for  this  technical  initiator  (or 
his  group)  prior  to  the  issuance  of  the  R.F.P.?"  As  shown  in  Table  1  61 
per  cent  of  the  winners  had  this  contractual  background,  while  only  34  per 
cent  of  the  losers  qualified  in  this  respect,  a  two  to  one  difference  in 
experience  of  the  two  groups  of  competitors.   A  number  of  the  respondents 
had  worked  on  more  than  one  contract  for  the  organization  studied. 


Some  initial  analyses  of  bidder-no  bidder  and  winner-loser  differences 
are  included  in  a  thesis  performed  by  one  of  the  author's  graduate 
research  assistants:   Lewis  G.  Pringle,  "An  Investigation  of  R&D  Marketing 
Strategy"  (unpublished  Master  of  Science  thesis,  M.I.T,  Sloan  School  of 
Management,  1965). 


Table  1.   Prior  Contractual  Experience 


Prior 

No  Prior 

Total 

Contract (s) 

Contract 

Responses 

Winners 

15 

9 

24 

Losers 

26 

51 

77 

Of  course,  background  contractual  experience  with  other  organizations 
or  In-house  Independent  research  might  also  qualify  the  competing  organiza- 
tions. For  this  more  general  area'  of  inquiry  the  questionnaires  sought 
data  on  effort  expended  "in  areas  of  Immediate  technical  relevance  to  the 
subject  procurement,  prior  to  receipt  of  the  R.F.P.".  Here  too  the  winners' 
experience  outdistanced  that  of  the  losers,  the  average  winner  having  had 
18.0  technical  man-years  of  prior  effort  in  contrast  with  the  11.2  man-years 
average  of  the  losers. 

Winners  have  apparently  been  in  much  closer  contact  with  the  govern- 
ment technical  group,  exchanging  ideas  by  formal  and  informal  means.   A 
two  to  one  difference  again  shows  up  between  winners  and  losers  in  their 
prior  submittal  of  unsolicited  proposals  to  the  same  customer  in  the  same 
technical  area,  with  32  per  cent  of  the  winners  and  15  per  cent  of  the  losers 
using  this  approach  to  technical  communication  and  technical  marketing. 

Table  2.   Unsolicited  Proposals 

Prior  Unsolicited        No  Prior  Total 

Proposals        Unsolicited  Proposals   Responses 

Winners  8  17  25 

Losers  12  67  79 


Apparently  R&D  competitors  learn  of  prospective  contracts  of  Interest 
by  means  other  than  the  formal  announcements.   Even  here,  however,  winners 
are  more  knowledgeable  than  losers,  with  71  per  cent  anticipating  the  R*F*P« 
versus  53  per  cent  of  the  losers. 

As  soon  as  an  R.F.P.  is  received  it  can  be  read  and  Judgments  made 
about  the  nature  of  the  competitive  situation.   Three  times  as  many  losers 
as  winners  thought  "the  procurement  appear^d^to  'belong'  to  someone  else", 
with  24  per  cent  or  one  of  every  four  losers  entering  a  competttion  he  be- 
lieved "wired"  against  him.   The  response  figures  are  shown  in  Table  3. 

Table  3.   Procurement  "Belonged"  to  Someone  Else 

Procurement       Procurement       Total 
"Belonged"     Did  Not  "Belong"    Responses 

Winners         2  23  25  * 

Losers         19  59  78 

On  the  other  hand,  four  times  as  many  winners  as  losers,  36  per  cent 
compared  with  nine  per  cent,  felt  the  procurement  "belonged"  to  their  own 
firm.  It  is  interesting,  if  perhaps  only  coincidental,  that  eight  per  cent 
of  the  winners  and  nine  per  cent  of  the  losers  were  "upset"  in  their  pre- 
dictions of  "wired  procurements.  Other  non-statistical  evidences  suggest 
that  both  groups  may  have  been  right,  but  that  new  factors  arose  during  the 
competitions  to  reverse  prior  government  preferences. 

Winners  of  research  and  development  contracts  apparently  had  better 
personal  relationships  with  government  technical  Initiators  than  did  losers, 
or  at  least  they  were  more  conscientious  of  the  importance  of  personal  con- 
tacts.  Slightly  more  of  them  knew  the  identity  of  the  technical  initiator 
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Table  4.   Procurement    "Belonged"  to  Own  Firm 


Procurement       Procurement       Total 
"Belonged"     Did  Not  "Belong"    Responses 


Winners 
Losers 


9 
7 


16 
70 


25 
77 


of  the  procurement,  and  relatively  more  o£  the  winners  established  contact 
with  government  technical  personnel  after  receipt  of  the  R.F.P*  but  before 
proposal  submittal. 

Table  5.   Knowledge  of  and  Contact  with  Technical  Initiator 


Knew 

Did  Not  Know 

Contacted 

No  Contact 

Initiator 

Initiator 

Initiator 

with  Initiator 

Winners 

20 

5 

12 

13 

Losers 

56 

23 

27 

51 

Given  these  indications  it  is  not  surprising  that  winners  had  more 
confidence  in  themselves  as  having  an  "advantage  over  a  hypothetical  company 
of  equal  technical  competence  whose  knowledge  of  customer  requirements  was 
limited  solely  to  information  contained  in  the  R.F.P."  The  answers  to  this 
question  suggest  that  winners  not  only  knew  information  not  contained  in 
the  R.F.P.  but  also  that  they  regarded  their  Incremental  knowledge  as  im- 
portant. 

The  nSiD  winners  also  thought  the  prospective  Jobs  were  Important  to 
their  government  customers  and  that  contracts  would  in  fact  be  awarded 
(rather  than  be  lost  In  the  red  tape)  and  result  in  significant  follow-on. 
When  asked  to  estimate  the  priority  that  the  customer  attached  to  the  pro- 


Not 

Total 

Advantaged 

Advantaged 

Response 

winners 

16 

7 

23 

Losers 

34 

41 

75 

curement,  the  winners  relative  to  the  losers  regarded  the  procurements  as 
higher  In  priority,  the  difference  In  their  answers  being  highly  signifi- 
cant (I.e.,  at  the  2  per  cent  level  of  probable  occurrence).   Furthermore, 
the  average  winner's  estimate  at  the  time  of  bidding  of  the  follow-on  to 
the  procurement  was  20  per  cent  higher  than  the  average  loser's  estimate. 
Getting  closer  to  a  technical  area  of  work  changes  a  winner's  views  of  Its 
Importance,  the  winner's  current  average  estimate  of  follow-on  being  double 
his  Initial  estimate,  while  the  average  loser's  estimate  has  been  unchanged. 

During  the  proposal  preparation  phase  the  winners  of  research  and 
development  contracts  also  showed  themselves  to  be  significantly  different 
from  losers.   68  per  cent  of  the  winners  versus  only  33  per  cent  of  the 
losers  stated  they  designed  the  technical  approach  of  their  proposals  to 
satisfy  known  technical  preferences  of  the  customer.   Furthermore,  28  per- 
cent of  the  winners  versus  14  per  cent  of  the  losers  directed  the  content 
of  their  proposals  toward  particular  individuals  In  the  government  agency. 

Table  7.   Pr6posal  Reflected  Preferences  of  and  Directed  to  Customer 


Reflected         Did  Not 

Customer  Reflect  Customer 
Preferences     Preferences 

Winners      17  8 

Losers       26  52 


Directed  at  Not  Directed 

Particular  at  Particular 

Individuals  Individuals 

7  18 

11  68 
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These  data  support  the  notion  that  his  more  extensive  prior  contacts  are 
crucial  to  differentiation  of  the  winner  from  the  loser. 

The  final  distinguishing  characteristic  of  the  R&D  winner  is  that  he 
does  not  pay  as  much  attention  as  do  losers  to  the  formal  aspects  of  pro- 
posal preparation.   A  recent  study  by  Thomas  J.  Allen  of  the  M.I.T.  Sloan 
School  of  Management  revealed  that  in  22  R&D  proposal  competitions,  the 
company  that  received  the  highest  technical  evaluation  (and  usually  won 
the  contract)  had  invested  on  the  average  less  technical  man-hours  in  the 
formal  proposal  than  did  the  next  four  highest  ranking  contenders  for  the 
award.  Furthermore,  his  study  showed  that  higher  ranking  companies  used 
less  outside  technical  consulting  than  did  lower  firms. ^  Among  respondents 
to  the  author's  questionnaires  only  16  per  cent  of  the  winners  used  tech- 
nical writers  to  improve  their  proposals  whereas  36  per  cent  of  the  losers 
resorted  to  this  form  of  superficial  competition. 

Table  8.   Use  of  Technical  Writers 


Vechnical 

Mo 

Technical 

Total 

Writers 

Writers 

Responses 

Winners 

4 

21 

25 

Losers 

29 

51 

80 

Thus,  the  data  indicate  that  companies  that  win  research  and  develop* 
ment  contracts  possess  more  prior  contract  and  other  relevant  technical 


2  Thomas  J.  Allen,  "The  R&D  Proposal  Preparation  Process:  Factors  Influ- 
encing Technical  Quality  (unpublished  manuscript,  M»I*T»  Sloan  School 
of  Management,  July,  1964),  p. 8. 
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experience  than  their  competitors,  have  prepared  more  related  unsolicited 
proposals,  understand  and  more  strongly  appreciate  the  government's  pro- 
posed programs  and  technical  preferences,  et  cetera,  et  cetera,  et  cetera! 

What  Determines  R&D  Awards? 

The  second  question  listed  at  the  outset  of  this  paper  inquired  about 
the  key  determinants  of  contract  awards  in  research  and  development.   As 
sources  of  data  pertaining  to  this  question  the  author  has  studied  100 
research  and  development  awards  made  by  one  NASA  and  two  Department  of  De- 
fense installations.   The  NASA  center  was  studied  in  1962  at  which  time 
ten  large  awards  (ranging  from  $1  million  to  $40  million  in  Initial  size) 
were  studied  by  file  search  and  open-ended  interviews  with  technical,  ad- 
ministrative, and  procurement  personnel.   These  large  contracts  were  inter- 
esting to  follow  and  furnished  an  informative  but  inconclusive  picture  of 
what  happens  in  the  award  of  R&D  contracts.   Three  faults  were  apparent 
in  the  study  approach:   (1)  the  size  of  the  awards  examined  necessitated 
a  small  yet  complex  sample;   (2)  the  newness  of  the  agency  produced  unique*- 
ness  of  procurement  techniques  and/or  policy  in  almost  every  case;   (3)  the 
emphasis  on  open-ended  interviews  led  to  rich  background  material  and  anec- 
dotal evidences  but  gave  only  sparse  persuasive  quantitative  results. 

With  this  background  it  was  decided  to  solicit  cooperation  from  var- 
ious Department  of  Defense  organizations,  working  with  contracts  of  $100,000 
and  more,  devoting  much  attention  to  the  gathering  of  quantitative  eviden- 
ces on  the  award  process.   The  first  two  DOD  organizations  contacted  agreed 
to  participate  in  the  research  and  made  available  their  procurement  files 
for  study  and  their  technical  and  procurement  personnel  for  Interviews. 
Without  the  complete  cooperation  of  these  organisations,  the  research  re* 
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ported  In  this  paper  would  not  have  been  possible.   Their  assistance  Is 
deeply  appreciated  although  for  obvious  reasons  their  anonymity  must  be 
preserved • 

In  the  first  of  these  Installations  forty-one  competitions  were 
studied  that  resulted  In  awards  of  research  and  development  contracts 
ranging  from  $100,000,  the  minimum  award  size  Included  In  the  study,  up 
to  eight  million  dollars  In  size.   In  this  first  Installation,  the  awards 
were  made  during  the  period  from  January,  1960,  to  June  of  1963.   The  se- 
cond study  In  another  DOD  field  center  covered  forty-nine  contract  awards, 
ranging  from  $100,000  to  In  this  case  $2,000,000  In  Initial  size;  and  these 
contracts  were  Issued  In  the  time  period  from  May,  1962,  to  June,  196A,  the 
time  that  data  gathering  was  Initiated  on  this  last  study.   Thus  the  data 
are  drawn  from  fairly  current  cases,  the  latter  study  reflecting  whatever 
Influences  the  McNamara  regime  has  had  on  the  R&D  award  process. 

The  next  five  figures  describe  some  of  Che  results  of  these  studies. 
Four  of  them  show  data  from  both  Defense  Department  contracting  organiza- 
tions, the  evidences  from  the  first  installation  at  the  top  of  the  page, 
and  organization  two  down  below.   The  award  structures  in  the  two  instal- 
lations studied  are  compared  on  the  basis  of  several  different  dimensions. 
Figure  1  shows  the  measure  that  Is  generally  regarded  as  the  most  Impor- 
tant determinant  of  R&D  awards,  I.e.,  the  evaluated  technical  rank  of  the 
competing  companies.   The  curves  are  frequency  distributions  of  the  number 
of  contracts  awarded  as  a  function  of  the  evaluated  technical  rank  of 
the  individual  award-winning  companies.   In  the  first  organization,  the 
graph  indicates  that  36  contracts  out  of  the  41  went  to  the  highest  tech- 
nically-ranked company.   Two  awards  went  to  organizations  ranked  i^two  tech- 
nically, and  so  forth.   In  organization  if2   where  49  awards  were  studied. 
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41  o£  the  awards  wenC  Co  the  highest  technically-ranked  organization, 
half  a  dozen  went  to  the  second  highest  ranked,  one  went  to  the  third 
highest  ranked.  The  technical  rank  data  appear  to  support  the  generally 
accepted  notion  that  In  research  and  development  the  technical  evaluation 
Is  the  key  Influence  on  contract  awards.   With  more  empirical  evidence  yet 
to  be  presented,  however,  it  may  be  wise  to  reserve  final  Judgment. 

It  is  clear  from  Figure  1  that  not  all  awards  go  to  the  highest  tech- 
nically-ranked company.   Some  of  the  awards  apparently  are  based  upon 
other  factors,  and  cost  considerations  are  suggested  Immediately.   However, 
before  cost  or  other  influences  can  affect  an  award,  competing  companies 
must  be  qualified  as  technically  acceptable.   Strict  adherence  to  the  use 
of  technical  acceptance  evaluations  varies  from  one  government  organization 
to  another.   The  second  DOD  center  examined  is  committed  to  the  practice 
of  first  qualifying  proposals  for  technical  acceptability  and  then  examin- 
ing the  cost  proposals  of  only  those  firms  that  are  technically  acceptable. 
The  same  people  who  do  the  technical  evaluation  determine  technical  accept- 
ability, of  course.   The  chart  of  Figure  2  demonstrates  that  in  16  cases  of 
the  49  awards  investigated  only  one  company  was  regarded  as  technically  accept- 
able. After  each  such  determination,  the  award  was  evldent--lt  either  went 
to  that  "acceptable"  company  or  the  job  might  be  cancelled  by  the  Procure- 
ment Office.   Contract  award,  however,  was  a  precondition  for  inclusion  in 
the  sample.   Therefore,  Figure  2  is  indicating  that  in  16  out  of  49  cases, 
the  only  influence  on  the  award  was  the  technical  assessment.   Going  further, 
there  are  19  cases  in  which  two  companies  survived  the  technical  assessment 
and  were  regarded  as  technically  acceptable.   This  now  gives  35  awards  out 
of  49,  about  70  per  cent  of  the  awards,  in  which  the  procurement  officer 
might  have  considered  no  more  than  one  or  two  companies.   The  next 
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point  on  the  curve  represents  an  additional  ten  cases  in  which  three  com- 
panies were  listed  as  technically  acceptable.  Thus,  after  the  technical 
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evaluation  in  45  out  of  49  award  cases,  better  than  90  percent,  no  more 
than  three  companies  were  still  in  the  running  for  the  award  of  the  R&D 
contract.   These  data  demonstrate  Che  power  of  the  technical  phase  of 
evaluation  and  of  the  technical  evaluators  in  determining  the  award 
recipient. 

Turning  now  to  other  possible  influences  upon  R&D  awards.  Figure  3 
illustrates  what  happens  when  the  award  winners  are  ranked  on  the  basis 
of  their  cost  positions,  looking  at  how  much  each  winner  bid  relative  to 
his  competitors  for  the  award.   It  is  Important  to  note  that  all  the  awards 
were  either  CFFF,  CPIF,  cost  reimbursable  or  cost  sharing  in  some  sense. 
Therefore,  in  no  case  was  there  a  formal  incentive  for  a  low  bid,  there 
being  no  fixed  price  competitions  in  all  of  Che  90  awards  studied.   It  is 
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well  known  that  the  typical  cost  growth  that  occurs  In  cost-plus  con- 
tracting Is  sufficient  wholly  to  distort  any  Initial  bids  that  are  presented. 
Yet,  looking  at  all  the  proposals  In  organization  fl    (even  those  that  were 
not  technically  qualified)  and  at  the  technically  acceptable  proposals  In 
organization  iH    (cost  data  were  not  kept  on  file  for  those  not  regarded  as 
technically  acceptable),  the  possibly  surprising  result  Is  that  the  low 
bidder  received  more  awards  than  any  other  bidder.   This  shows  up  even  though 
for  the  second  group  studied  the  sixteen  cases  were  omitted  In  which  only 
one  company  was  ranked  acceptable.   (It  was  not  known  whether  to  treat 
that  one  company  as  high  bidder  or  low  bidder.) 

These  results  still  do  not  fully  explain  the  R&D  award  process.   First 
It  has  been  shown  that  technical  evaluation  appears  to  determine  many  of 
the  awards;  secondly,  cost  evaluation  apparently  explains  other  awards. 
But  there  Is  one  catch!   All  of  these  formal  evaluations,  of  course,  occur 
after  proposals  have  been  solicited,  after  proposals  have  been  prepared, 
after  proposals  have  been  received  by  the  government  agency.   But  some  things 
do  happen  even  before  these  phases.   And  one  of  the  events  that  precedes 
proposal  solicitation,  preparation,  receipt,  and  evaluation  Is  that  the 
technical  Initiator  In  the  government  agency  prepares  a  procurement  request 
(PR).   On  this  PR  he  indicates,  among  other  things,  a  list  of  suggested 
companies  that  he  has  In  mind  for  doing  the  Job.   In  the  research  conducted 
at  both  DOD  organizations  the  procurement  request  forms  In  the  government 
flics  were  surveyed  andllod  to  the  results  shown  In  the  next  figure.  For 
both  agencies  the  plots  show  the  frequency  of  awards  as  a  function  of  the 
position  of  the  winning  company's  name  In  the  list  of  suggested  companies 
on  the  procurement  request  form.   It  should  be  noted  Chat  these  lists  were 
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prepared  by  the  technical  initiator  of  the  procurement  six  months  to  a 
year  prior  to  the  evaluation  of  proposals.  Alphabetical  lists  were  neces- 
sarily excluded  from  this  analysis.  (There  Is  little  reason  to  believe 
that  a  company  name  beginning  with  "A"  Is  more  favorable  than  one  starting 
with  "Z"  for  getting  government  business.)  Looking  at  these  non-alphabeti- 
cal cases,  a  very  simple  hypothesis  Is  proposed;  I.e.,  where  a  firm's  name 
appears  on  the  original  suggested  source  list  prepared  by  the  technical  in- 
itiator Is  a  good  Indicator  of  the  initiator's  preferences.   When  one  plots, 
as  Is  done  In  Figure  4,  the  award  of  R&D  contracts  as  a  function  of  listed 
position  on  the  PR  form,  it  appears  that  Indeed  the  data  support  this  hypo- 
thesis in  both  DOD  organizations. 

One  more  step  can  be  taken  with  this  PR  listings  data.  If  a  company 
preferred  by  the  technical  initiator  (and  so  listed)  did  not  bid  (there 
appear  to  be  numerous  slips  of  this  sort),  it  Is  obvious  that  this  pre- 
ferred company  could  not  win.   Amending  the  lists  by  dropping  these  no- 
bidders,  the  frequency  of  awards  as  a  function  of  the  revised  positions  of 
those  who  were  listed  and  did  bid  produces  the  sharper  curves  of  Figure  5. 

It  seems  obvious  that  the  award  Indicators  shown  in  Figures  4  and  5 
are  important.   The  data  presented  are  drawn  from  procurement  request 
forms  that  precede  by  six  months  to  a  year  the  technical  evaluation  measures 
shown  in  Figure  1.   It  is  almost  always  the  government  engineer  or  scientist 
who  prepares  the  lists  Incorporated  In  Figures  4  and  5  who  also  prepares 
the  technical  evaluations.   Whatever  produces  the  technical  prejudice,  the 
feelings  of  confidence  and  tcust  In  one  particular  organization  rather  than 
in  another,  that  is  built  into  the  initiator  at  the  time  that  he  prepares 
the  procurement  request,  seems  to  stick  with  him  throughout  the  entire  for- 
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mal  competitive  process.   It  is  apparent  that  the  determinants  of  awards 
of  research  and  development  contracts  are  Influences  on  the  Inltiator- 
evaluator  prior  to  the  preparation  of  the  procurement  requests,  not  dur- 
ing the  period  of  time  of  formal  proposal  solicitation,  proposal  preparaft'. 
tion,  and  proposal  evaluation.   The  proposal  solicitation,  preparation, 
and  evaluation  are  responses  to  a  decision  by  the  technical  initiator  to 
undertake  a  set  of  technical  acts  under  contract,   It  is  clear  that  he 
generally  enters  into  that  set  of  acts  already  committed,  at  least  in  his 
own  mind,  to  one  or  two  companies.   Other  evidences  exist,  primarily  from 
interviews  with  the  government  technical  and  procurement  staffs,  that  sup- 
port this  conclusion. 
V?hat  are  the  Benefits  of  the  Present  System? 

Given  the  conclusions  now  reached  on  the  characteristics  of  award 
winners  and  of  the  award  process,  it  is  appropriate  to  pose  more  direct 
questions  of  cost/benefits  comparison.   The  present  R&D  procurement  system 
features  the  formal  solicitation  and  evaluation  of  formal  proposals  from 
Industry.   Four  possible  benefits  of  this  approach  haVe  been  proposed:   (1) 
better  technical  ideas  are  obtained  by  competitive  solicitation;   (2)  lower 
costs  are  secured  for  the  government;   (3)  in  general,  "objectivity",  re- 
garded as  "good"  for  its  own  sake,  prevails;  and  (A)  the  system  is  "demo- 
cratic", open  to  all  new  comers.    Each  of  these  presumed  benefits  is 
questionable. 

1.  It  is  true  that  some  better  technical  ideas  are  obtained  by  com- 
petitive solicitation.   This  is  shown  by  the  fact  that  seven  of  the  41 
awards  in  the  first  DOD  Installation  went  to  companies  not  originally  re- 
commended by  the  government  technical  initiator.  Only  two  awards  of  organ- 
ization .  #2  went  Co  non-recommended  firms,  and  one  of  these  happened  in 
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response  to  the  strong  personal  preferences  of  a  newly-assigned  technical 
evaluator  who  replaced  the  original  initiator.  Certainly  some  of  these 
awards  were  made  because  of  unique  techtilcal  ideas  proposed  by  these  winners. 
However,  most  of  the  award  winners  were  preferred  on  technical  grounds  long 
in  advance  of  the  formal  phases  of  the  process.   Thus,  the  obtaining  of 
their  Ideas  is  not  a  resultant  of  the  formally  competitive  aspect  of  pro- 
curement.  Furthermore,  the  differences  between  Figure  4  and  S  reflect  a 
number  of  no-bid  decisions  by  companies  initially  preferred  by  the  technical 
initiators  of  the  procurements.   There  is  good  likelihood  that  a  number  of 
the  better  technical  ideas  were  lost  to  the  government  because  the  better 
firms  were  not  responsive  to  large-scale  open  solicitation. 

2.  There  is  no  way  of  evaluating  whether  or  not  lower  bids  are  secured 
by  competitive  solicitation.   In  contrast  with  procurement  of  off-the-shelf 
hardware,  specifications  for  RSd)  projects  are  constantly  changing  and  no 
tvfo  procurements  (except  for  parallel  studies)  are  alike.   Therefore,  it 
is  impossible  to  compare  contract  amounts  obtained  by  different  procure- 
ment methods.   Furthermore,  with  the  large  cost  growth  that  typically  oc- 
curs in  research  and  development  reliance  upon  initial  bids  X^r  on  cost 
growth)  is  not  a  convincing  measure. 

Two  results  of  the  studies  shed  some  insight  into  the  Question  of  con- 
tract cost.   First,  the  evaluation  reports  and  the  interviews  in  DOD  organ- 
ization #1  showed  conclusively  that  at  least  four  awards  were  made  because 
of  cost  differences  in  proposals,  not  because  of  better  technical  approaches. 
In  two  of  these  four  cases,  the  assumed  cost  benefits  turned  out  to  have 
been  grossly  misleading,  and  the  government  project  monitors  regretted  the 
awards  to  these  lower  bidders.  Thus,  it  is  possible  that  lower  bid  costs 
are  obtained  occasionally  through  competitive  solicitation,  but  that  higher 
final  oosts  often,  result <> 
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Secondly,  an  important  observation  is  that  the  originally  preferred 
companies,  that  later  become  the  higher  technically  evaluated  companies, 
also  are  most  often  the  lower  bidders,  as  shovm  by  the  earlier  figures 
presented.   It  has  been  suggested  by  both  government  and  industry  officials 
that  this  phenomenon  is  due  to  the  "Insider's"  better  understanding  of  what 
is  really  wanted  by  the  government,  resulting  in  a  lack  of  attempt  to  cover 
all  possibilities  in  his  proposed  solution  approach.   Whether  or  not  this 
explanation  is  valid,  it  is  true  that  the  lower  bidders  (after  the  fact  of 
solicitation)  are  the  same  firms  that  would  have  received  awards  had  Initial 
government  Initiator  preferences  been  followed  by  a  more  direct  form  of  con- 
tract award. 

3.  A  major  presumption  is  that  "objectivity",  viewed  by  many  as  good 
for  its  own  sake,  results  from  the  widespread  solicitation,  formal  eval- 
uation, multi-level  review  process  used  currently  for  R&D  procurement. 

That  this  argument  is  patently  false  was  demonstrated  in  an  earlier  article 

3 
by  the  author.   Most  evaluation  teams  are  dominated  by  one  or  two  indivi- 
duals, their  evaluations  are  dominated  by  experiences  that  precede  the 
readings  of  the  proposals,  and  the  committees  appointed  to  review  their  work 
usually  rubber  stamp  the  earlier  reports. 

A.  The  final  benefit  often  claimed  for  the  present  procurement  system 
is  that  it  is  "democratic",  permitting  new  companies  to  break  into  the  R/.D 
contracting  business.   It  appears  from  the  evidences  already  presented 
that  new  companies  can  win  awards,  but  not  primarily  because  of  the  formal 


3 
Edward  B.  Roberts,  "How  the  UoS.  Buys  Research",  International  Science  and 
Technology,  no.  33,  September,  1964,  pp.  70-77. 
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aspecCs  of  the  competitions.   Figures  4  and  5  show  that  the  award  winners 
are  generally  known  before  the  formal  phases  get  underway.  The  answers 
to  the  first  question  posed  In  this  paper  Indicate  that  award  winners  have 
done  extensive  prior  contact  work  with  the  government  Initiators.   It  is 
this  personal  contact  route,  n6t  the  responses  to  formal  solicitations, 
that  leads  to  R&D  awards  for  old  companies  as  well  as  new. 

What  does  the  System  Cost? 

Four  types  of  costs  result  from  the  present  R&D  procurement  system 
that  might  be  avoided  under  an  alternative  method.   They  Include:  (1) 
absolute  dollar  outlay  for  proposal  preparation  and  evaluation;  (2)  mls- 
allocatlon  of  government  and  Industry  technical  skills;  (3)  added  time 
delays  In  the  achievement  of  desired  R&D  results;  (4)  lowering  of  the  ethi- 
cal standards  of  participants  In  the  Industry.   Complete  analyses  of  the 
cost  aspects  of  the  government  and  Industry  data  presented  earlier  are  not 
yet  available.   Thus,  the  comments  here  will  be  limited  to  first-order 
effects. 

1.  Companies  expend  and  the  government  reimburses  through  overhead 
allowances  large  amounts  for  the  direct  costs  of  preparing  R&D  proposals. 
The  total  costs  depend  U{>pn  the  number  of  firms  bidding  on  a  contract  and 
the  number  of  bidders  Is  dependent  In  turn  on  the  number  solicited  to  parti- 

clpate.   This  dependency  Is  shown  by  Figure  6  as  well  as  by  the  results  of 

4 
correlation  analysis.    In  this  regard  It  should  be  noted  that  In  both  DOD 

organizations  studied,  double  the  number  of  firms  were  solicited  than  were 


4 

The  Pearson  Product  Moment  Correlation  Coefficient  relating  number  of 

bidders  to  number  solicited  In  DOD  organization  #2  is  0.53,  with  a  con- 
fidence level  of  1  percent. 


C«4 


o 

o> 

o 

00 

o 

O 

0) 

r>. 

S 

a. 

M 

o 

U 

v£) 

M 

O 

hJ 

H 

o 

to 

CO 

O 

u 

u-» 

to 

to 

•z. 

M 

2 

vO 

O 

1 

CO 

« 

<»■ 

s 

V4 

3) 

O 

8 

s 

(14 

O 
CM 

8 

o 


\r\ 


u-» 


aaAiaoaH    sivsoaoHd 


•23- 


-24 
originally  suggested  by  the  technical  initiator o   This  high  percentage 
of  "add-'ons"  for  the  sake  of  competition  seems  an  Integral  part  of  the 
present  system.   It  need  be  remembered  that  hardly  any  of  these  "add-ons" 
win  the  contract,  but  they  do  help  increase  the  total  procurement  cost. 

2.  The  present  system  ties  up  the  best  industry  people  in  proposal 
preparation  efforts^  denying  their  skills  from  the  contracts  already  in- 
house  in  the  competing  firm.  The  most  competent  government  technical  per- 
sonnel are  similarly  engaged  in  proposal  evaluation  formalities.  Instead 
of  in  in-house  government  research. 

3.  On  the  average  six  to  eight  months  pass  from  preparation  of  Che 
initial  procurement  request  form  until  contract  award.   The  life  of  the 
typical  R&D  contract  is  only  one  to  two  years.   Thus,  the  present  pro- 
curement process  adds  from  one-fourth  to  one-half  the  scheduled  duration 
to  the  life  cycle  of  R&D  contracts  in  the  size  range  studied,  i.e.,  up 
to  about  $1  million.   With  the  award  usually  going  to  a  company  identi- 
fiable at  the  time  of  procurement  request,  serious  doubts  arise  as  to  the 
Justification  of  the  added  delay. 

4.  The  ethical  standards  of  both  industrial  and  government  parti- 
cipants in  the  R&D  procurement  are  lowered  by  the  pressures  of  the  pre- 
sent system.   Industrial  people  are  pushed  to  intentionally  low  b:'.d,  even 
when  they  know  that  costs  will  rise  orders  of  magnitude  during  contract 
life.   Industrial  people  see  and  respond  to  competitive  solicitations  that 
they  know  were  prepared  by  their  own  staff  as  unsolicited  proposals  months 
before.   Industry  often  responds  to  requests  for  proposals  only  to  help 
out  a  friend  in  government  who  Is  seeking  to  keep  the  procurement  office 
happy  about  the  "competitiveness"  of  the  procurement.   Government  people, 

on  the  other  hand,  are  led  to  devious  practices  to  ensure  awards  to  com- 
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panles  they  believe  competent  to  carry  out  the  RSsD  tasks.  None  of  these 
practices  are  designed  to  encourage  the  best  technical  people  to  stay  In 
the  R&D  contracting  business,  on  either  the  Industry  or  government  side. 

What  Alternatives  Exist? 

The  questions  and  answers  presented  in  this  paper  are  difficult  to 
evaluate  conclusively.   Whether  or  not  the  present  system  provides  any  real 
benefits  is  unresolved.   How  expensive  these  benefits  should  be  is  also  a 
value- laden  question.   In  the  opinion  of  the  author  the  present  R&D  pro- 
curement system  unnecessarily  stresses  formalities  of  competition,  when 
Intense  person-to-person  informal  competition  seems  to  be  having  the 
greatest  real  effects.   Why  should  so  much  time,  cost,  and  technical  skills 
be  wasted  in  Improper  emphasis  on  formal  proposal  preparation  and  evalua- 
tion?  In  this  final  section  are  presented  some  changes  in  attitudes  and 
policies  that  should  strengthen  the  effectiveness  of  R&D  procurement.^ 
These  changes  appear  to  be  most  suitable  for  R&D  projects  involving  less 
than  $1  million  of  contract  funding. 

1.   Government  technical  initiators  should  be  given  greater  flexibility 
in  the  form  of  information-seeking  that  they  may  use  prior  to  R&D  awards. 
Alternate  schemes  should  be  encouraged,  as  appropriate  to  each  Individual 
procurement,  including  wider  use  of  limited-source  solicitations  of  the 
top  two  or  three  technically  preferred  companies,  the  use  of  oral  proposals, 
laboratory  visits  by  government  Initiators  to  assess  both  facilities  and 
technical  staff,  question-answer  proposals  and  any  other  approach  that 
reduces  the  red  tape  and  gives  the  government  evaluator  the  information 
needed  for  an  award  decision. 


^The  changes  were  originally  documented  In  the  author's  article,  "Improving 
R&D  Procurement",  The  Scanner.  Fall,  1965. 
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2.  Needless  practices  of  a  few  government  agencies  should  be  weeded 
out.   In  an  apparent  attempt  to  keep  Industry  on  the  "outs",  some  Instal- 
lations go  to  great  effort  to  keep  procurement  details  secret.   Occasion- 
ally one  government  installation  sends  out  the  RFPs  and  then  tricks  In- 
dustry by  forwarding  the  proposals  to  a  second  center  for  evaluation. 
These  practices  reflect  a  mood  of  warfare  between  government  and  Industry, 
Instead  of  the  cooperative  partnership  that  Is  needed  In  pioneering  tech- 
nical achievements. 

3.  Changes  In  cost-allowances  should  be  considered  that  will  en- 
courage contractors  to  shift  much  of  the  present  relatively  unproductive 
proposal  budgets  Into  more  advanced  In-house  R&D,  and  Into  support  of 
extended  technical  publication  and  communication  programs  on  the  companies' 
progress  In  research  studies.   In  line  with  such  a  shift.  It  Is  probably 
advisable  to  discourage  sometimes  Irreaponsible  "bllnd"-bldding  practices 
by  companies  that  only  add  to  the  government  evaluators'  burdens.   Cost 
allowances  for  proposals  might  be  restricted  to  "acceptable"  bidders  or 

to  the  top  few  companies,  or  a  sliding  scale  of  cost  reimbursements  might 
be  based  on  the  technical  evaluators'  rankings. 

A.  To  further  Improve  the  communication  between  government  and 
Industry  of  technical  requirements  and  technical  progress  ,  some  of  the 
time  released  by  reduction  of  proposal  preparation  and  evaluation  work- 
loads should  be  invested  In  more  frequent  symposia  sponsored  cooperatively 
by  the  several  agencies  working  In  each  technical  field  for  exchanges  with 
Interested  contractors.   Such  stress  upon  Idea  and  progress  communication 
will  not  succeed,  however,  unless  companies  feel  confident  that  signifi- 
cant Ideas  might  readily  be  recognised  by  unhesltant  awards  of  sole-source 
contracts. 
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5.  Finally,  the  accomplishment  of  this  more  open,  more  responsible 
approach  to  research  and  development  contracting  demands  that  government 
technical  initiators  and  evaluators  include  more  of  the  best  people 'in 
each  technical  field.   In  order  to  accomplish  this, , programs  of  Job  en- 
largement need  be  undertaken  in  government  to  attract  and  hold  the  most 
competent  scientists,  engineers,  and  technical  administratbrs.   More 
explicit  recognition  of  status,  combined  with  higher  salaries,  particularly 
for  the  B&D  program  managers  in  both  civilian  and  military  categories,  is 
urgently  needed. 
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